Segment-based encoding system using segment hierarchies

ABSTRACT

Video data is compressed using segmentation of reference frames where the segmentation results in a hierarchy of segments. The hierarchy can be generated bottom-up, where a first set of segments is generated and then a second set of segments generated by grouping segments of the first set, based on pixel value and/or segment boundaries. The hierarchy can be generated top-down, where a first set is generated and then a second set is generated by dividing up segments of the first set, based on pixel value and/or segment boundaries. The hierarchy can be included in the compressed video data or omitted such that a decoder would have to independently generate it. The hierarchy can be two or more levels. The hierarchy can be used for editing, formatting and/or compressing frames, as well as associating metadata with elements of the frame and coding motion, residue or other kinetic information.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.09/550,705, filed on Apr. 17, 2000, now U.S. Pat. No. 6,600,786, whichis a non-provisional of and claims priority from U.S. application Ser.No. 60/129,854, filed on Apr. 17, 1999 and U.S. application Ser. No.60/129,853, filed on Apr. 17, 1999. The complete disclosures of theseapplications are incorporated herein by reference for all purposes.

FIELD OF THE INVENTION

The present invention relates generally to the compression of videodata, and more particularly to a synchronized encoder/decoder systemwherein the decoder performs a segmentation that is performed by theencoder for encoding video data so that the encoder need not convey tothe decoder the entire results of the encoder's segmentation.

BACKGROUND OF THE INVENTION

1. Brief Introduction

As more communication requires video, such as real-time streaming ofvideo, video conferencing, digital television, interactive televisionand Internet-based communications such as hypertext transport of WorldWide Web (WWW) content, more efficient ways of utilizing existingbandwidth are needed. This is because the typical bandwidth allocated toa particular transmission mode (e.g., broadcast, cable, telephone lines,etc.) is much less than the bandwidth typically required for a videostream. Thus, if such modes are to carry video, compression is needed.Compression is also needed where the video is stored, so that storagecapacity is efficiently used. The advent of multi-media capabilities onmost computer systems has taxed traditional storage devices, such ashard drives, to their limits.

Compression allows digitized video sequences to be representedefficiently, allowing more video to be transmitted in a given amount oftime over a given channel, or more video to be stored in a given storagemedium. Compression does this by reducing the bitstream, or videoinformation flow, of the video sequences at a transmitter (which can beplacing the bitstream into a channel or storing into a storage medium)while retaining enough information that a decoder or receiver at theother end of the channel or reading the storage medium can reconstructthe video in a manner adequate for the specific application, such astelevision, videoconferencing, etc.

Video is typically represented by a sequence of images, called “frames”or “video frames” that, when played in sequence, present the video. Asused herein, a video stream might refer to a video and audio stream,where the audio is included with the video. However, for simplicity,just the video compression is often described.

As the terms are used herein, an image is data derived from amulti-dimensional signal. The signal might be originated or generatedeither naturally or artificially. This multi-dimensional signal (wherethe dimension could be one, two, three, or more) may be represented asan array of pixel color values such that pixels placed in an array andcolored according to each pixel's color value would represent the image.Each pixel has a location and can be thought of as being a point at thatlocation or as a shape that fills the area around the pixel such thatany point within the image is considered to be “in” a pixel's area orconsidered to be part of the pixel. The image itself might be amultidimensional pixel array on a display, on a printed page, an arraystored in memory, or a data signal being transmitted and representingthe image. The multidimensional pixel array can be a two-dimensionalarray for a two-dimensional image, a three-dimensional array for athree-dimensional image, or some other number of dimensions.

The image can be an image of a physical space or plane or an image of asimulated and/or computer-generated space or plane. In the computergraphic arts, a common image is a two-dimensional view of acomputer-generated three-dimensional space (such as a geometric model ofobjects and light sources in a three-space). An image can be a singleimage or one of a plurality of images that, when arranged in a suitabletime order, form a moving image, herein referred to as a video sequence.

Pixel color values can be selected from any number of pixel colorspaces. One color space in common use is known as the YUV color space,wherein a pixel color value is described by the triple (Y, U, V), wherethe Y component refers to a grayscale intensity or luminance, and U andV refer to two chrominance components. The YUV color space is commonlyseen in television applications. Another common color space is referredto as the RGB color space, wherein R, G and B refer to the Red, Greenand Blue color components, respectively. The RGB color space is commonlyseen in computer graphics representations, along with CYMB (cyan,yellow, magenta, and black) often used with computer printers.

Video compression is possible because an uncompressed video sequencecontains redundancies and some of the video signal can be discardedwithout greatly affecting the resulting video. For example, each frameof a video sequence representing a stationary scene would be nearlyidentical to other frames in the video sequence. Most video compressionroutines attempt to remove the superfluous information so that therelated image frames can be represented in terms of previous imageframe(s), thus eliminating the need to transmit an entire image for eachvideo frame. Alternatively, routines like motion JPEG, code each videoframe separately and ignore temporal redundancy.

2. Known Compression Techniques

There have been numerous attempts at adequately compressing videoimagery. These methods generally fall into the following twocategories: 1) spatial redundancy reduction, and 2) temporal redundancyreduction.

2.1. Spatial Redundancy Reduction

Spatial redundancy reduction takes advantage of the correlation amongneighboring pixels in order to derive a more efficient representation ofthe important information in an image frame. These methods are moreappropriately termed still-image compression routines, as they generallyaddress each frame in isolation, i.e., independent of other frames inthe sequence. Because of this, they do not attempt to temporal, orframe-to-frame, redundancy. Common still-image compression schemesinclude JPEG, wavelets, and fractals.

2.1.1. JPEG/DCT Based Image Compression

One of the first commonly used methods of still-image compression wasthe direct cosine transformation (“DCT”) compression system, which is atthe heart of JPEG. DCT operates by representing each digital image frameas a series of cosine waves or frequencies and quantizing coefficientsof the cosine series. The higher frequency coefficients are quantizedmore harshly than those of the lower frequencies. The result of thequantization is a large number of zero coefficients, which can beencoded very efficiently. However, JPEG and similar compression schemesdo not address the crucial issue of temporal redundancy.

2.1.2. Wavelets

As a slight improvement to the DCT compression scheme, the wavelettransformation compression scheme was devised. This system is similar tothe DCT, differing mainly in that an image frame is represented as aseries of wavelets, or windowed oscillations, instead of as a series ofcosine waves.

2.1.3. Fractals

Another technique is known as fractal compression. The goal of fractalcompression is to take an image and determine a single function, or aset of functions, which fully describe(s) the image frame. A fractal isan object that is self-similar at different scales or resolutions, i.e.,no matter what resolution one looks at, the object remains the same. Intheory, where fractals allow simple equations to describe compleximages, very high compression ratios should be achievable.

Unfortunately, fractal compression is not a viable method of generalcompression. The high compression ratios are only achievable forspecially constructed images, and only with considerable help from aperson guiding the compression process. In addition, fractal compressionis very computationally intensive.

2.2. Temporal and Spatial Redundancy Reduction

Adequate motion video compression requires reduction of both temporaland spatial redundancies. Temporal redundancy can be reduced byreplacing all or part of the bits representing the image of a frame withone or more references to other frames or portions of a frame. Thisallows a small number of bits to represent a larger number of bits.Block matching is the basis for most currently used effective means oftemporal redundancy removal.

In block matching, an image frame is subdivided into uniform size blocks(more generally, into polygons), and each block is tracked from oneframe to another and represented by a motion vector, instead of havingthe block re-coded and placed into the bitstream for a second time.Examples of compression routines that use block matching include MPEGand variants thereof.

MPEG encodes the first frame in a sequence of related frames in itsentirety as a so-called intra-frame, or I-frame. An I-frame is a type ofkey frame, meaning an image frame that is completely self-contained andnot described in relation to any other image frame. To create anI-frame, MPEG performs a still-image compression on the frame, includingdividing the frame into 16 pixel by 16 pixel square blocks. Other(so-called “predicted”) frames are encoded with respect to the I-frameby predicting corresponding blocks of the other frame in relation tothat of the I-frame. That is, MPEG attempts to find each block of anI-frame within the other frame. For each block that still exists in theother frame, MPEG transmits the motion vector, or movement, of the blockalong with block identifying information. However, as a block moves fromframe to frame, it may change slightly. The difference relative to theI-frame is known as residue. Additionally, as blocks move, previouslyhidden areas may become visible for the first time. These previouslyhidden areas are also known as residue. That is, the collectiveremaining information after the block motion is sent is known as theresidue, which is coded using JPEG and included in the bitstream tocomplete the image frame.

Subsequent frames are predicted with respect to either the blocks of theI-frame or a preceding predicted frame. In addition, the prediction canbe bi-directional, i.e., with reference to both preceding and subsequentI-frames or predicted frames. The prediction process continues until anew key frame is inserted, at which point a new I-frame is encoded andthe process repeats itself.

Although state of the art, block matching is highly inefficient andfails to take advantage of the known general physical characteristics orother information inherent in the images. The block method is botharbitrary and crude, as the blocks do not have any relationship withreal objects in the image. A given block may comprise a part of anobject, a whole object, or even multiple dissimilar objects withunrelated motion. In addition, neighboring objects will often havesimilar motion. However, since blocks do not correspond to real objects,block-based systems cannot use this information to further reduce thebitstream.

Yet another major limitation of block-based matches arises because theresidue created by block-based matching is generally noisy and patchy.Thus, block-based residues do not lend themselves to good compressionvia standard image compression schemes such as DCT, wavelets, orfractals.

2.3. Alternatives

It is well recognized that the state of the art needs improvement,specifically in that the block-based method is extremely inefficient anddoes not produce an optimally compressed bitstream for motion videoinformation. To that end, the very latest compression schemes, such asMPEG4, allow for the inclusion of limited structural information, ifavailable, of selected items within the frames rather than merely usingarbitrary-sized blocks. While some compression gains are achieved, theassociated overhead information is substantially increased because, inaddition to the motion and residue information, these schemes requirethat structural or shape information for each object in a frame mustalso be sent to the receiver.

Additionally, as mentioned above, the current compression methods treatthe residue as just another image frame to be compressed by JPEG using afixed compression.

BRIEF SUMMARY OF THE INVENTION

Video data is compressed to form compressed video data wherein the videodata comprises a plurality of image frames including a set of referenceframes and a set of nonkey frames. The set of nonkey frames are encodedwith reference at least to a segmentation of at least one of the imageframes in the set of reference frames. The segmentation is an assignmentof some or all of the pixels of the reference frame to at least onesegment based on at least one of a color value or a location of thepixels in the reference frame. In an embodiment, video data is encoded afirst reference frame and segmenting the first reference frame into ahierarchy of segments. The hierarchy of segments can be generatedbottom-up, where a first set of segments is generated and then a secondset of segments is generated by grouping segments of the first set ofsegments, possibly repeating that step to form a hierarchy of three ormore levels. The grouping can be done based on pixel value and/orsegment boundaries.

Alternatively, the hierarchy of segments can be generated top-down,where a first set of segments is generated and then a second set ofsegments is generated by dividing up segments of the first set ofsegments, possibly repeating that step to form a hierarchy of three ormore levels. The division can be done based on pixel value and/orsegment boundaries. The hierarchy of segments can be included in thecompressed video data or omitted such that a decoder would have toindependently generate at least a portion of the hierarchy of segments.

In some embodiments, the hierarchy of segments can be used for editingframes, formatting frames and/or compressing frames. The hierarchy ofsegments is also usable for associating metadata with elements of theframe and/or coding one or more of motion, residue or other kineticinformation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a video stream processing system; FIG. 1(a) illustrates an example where video is compressed for transmissionover a channel; FIG. 1( b) illustrates an example where video iscompressed for storage.

FIG. 2 is a block diagram of an encoder according to embodiments of thepresent invention.

FIG. 3 is a diagram illustrating structure of a video stream accordingto embodiments of the present invention.

FIG. 4 is a diagram illustrating another variation of structure of avideo stream.

FIG. 5 is a block diagram of a decoder according to embodiments of thepresent invention.

FIG. 6 is a block diagram of a portion of an encoder, such as theencoder of FIG. 2, including a modeller.

FIG. 7 is an illustration of exposed areas.

FIG. 8 is a flowchart of an encoding process.

FIG. 9 is a flowchart of a decoding process.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a block diagram of a video stream processing system 10. System10 accepts video data from any number of sources, encodes it usingencoder 100 such that the video data is compressed (i.e., occupies fewerbits than the uncompressed video data) for transport or storage. System10 includes a decoder 200 that receives the transported or storedcompressed video data and decompresses for use by any number of videosinks (users).

Merely by way of example, possible video sources include a video camera,a video storage system (typically storing uncompressed, or partiallycompressed, video data), a high-speed channel, such as a cable link orbroadcast link capable of transmitting uncompressed or partiallycompressed video data, or a video player, such as a VCR or DVD player.Possible video users, for example, might include a display device, suchas a monitor or television, a video processor or video storage that canstore decoded video data.

FIG. 1( a) illustrates an example where video is compressed fortransmission over a channel 120. Channel 120, for example, could be adigital subscriber line (DSL), a cable modem, a dialup connection,broadcast, cable broadcast, satellite transmission, or the like. In suchcases, the video is compressed so that it can be transmitted usingavailable bandwidth efficiently

FIG. 1( b) illustrates an example of a system 20 where video iscompressed for storage. As shown, encoder 100 encodes video data forstorage in compressed video storage 130 for later retrieval by decoder200. Storage 130 might be, for example, a hard drive, a memory card, apersonal video recorder (PVR), RAM, CD, DVD, or any other suitablestorage.

Note that the same encoder and decoder can be used for a transmissionsystem as used for a storage system. Of course, the encoders anddecoders could be different. The differences could be external, such aschanging the output of the encoder to point to a storage device ratherthan a channel, but the changes could also be internal, such as changingthe methods used by the encoder depending on whether or not theencoder's output is time critical. For example, if it is known a priorithat the encoded video will not be read from storage right away, theencoder could trade off speed for improved compression.

In a basic operation, video data, usually uncompressed video data, isprovided to encoder 100, which encodes the video data to form compressedvideo data that occupies fewer bits than the uncompressed video data,and preferably much fewer bits, and makes the uncompressed video dataavailable to the decoder (via a channel, storage, or a combinationthereof). The decoder in turn decompresses the compressed video data toarrive at an exactor approximate copy of the uncompressed video dataprovided to the input of the encoder.

FIG. 2 is a block diagram of encoder 100 according to embodiments of thepresent invention. As shown there, encoder 100 comprises a frame loader202, a frame compressor 204, a motion matcher 206, a residue generator208, an output scheduler 210 and a segmenter 220. Also shown are storagefor data being processed, such as a frame buffer 230 for holding all orpart of a current frame, frame buffer 232 for holding all or part of areference frame, segment data set storage 234, kinetic informationstorage 236 for storing motion factors and other kinetic information,and residue data storage 238. Also shown, and explained below are aframe decompressor 240 and a frame regenerator 242.

Frame loader 202 is configured to receive uncompressed video in andprovide the uncompressed video in a frame-by-frame manner to framebuffer 230. It should be understood that the video in could be partiallycompressed and could be in any of the variety of formats. As shown,frame buffer 230 is coupled to frame processor 204, motion matcher 206,and residue generator 208 to provide all or part of the informationembodied in the current frame stored in frame buffer 230.

As used herein, the term “current frame” refers to a frame of videobeing processed by the encoder. In a typical operation, a frame isloaded into frame buffer 230 and becomes the current frame, that currentframe is processed and another frame is loaded into frame buffer 230 andthat frame would then be the current frame. The other frame buffer,frame buffer 232, is coupled to motion matcher 206 and residue generator208 to provide all of part of the information content of the referenceframe. Frame buffer 232 is also coupled to a segmenter 220, which is interm coupled to storage 234, thereby allowing segmenter 220 to generateand store a segment data set associated with the reference frame.Storage 234 is coupled to motion matcher 206 to allow motion matcher 206to obtain all or part of a segment data set.

Residue generator 208 is coupled to frame buffers 230 and 232, as wellas kinetic information storage 236 such that residue generator 208 canuse information stored therein to generate residue data stored inresidue storage 238.

As used herein, the term “reference frame” refers to a frame whoseinformation content is used, at least in part, in the encoding of thecurrent frame. In the general case, the current frame might be encodedwith reference to more than one reference frame, but for clarity encoder100 in its operation is described here where only one reference frame isneeded. As used herein, the term “key frame” refers to a frame that isencoded such that it can be decoded without reference to other frames.Note that reference frames are not required to be key frames but can beframes that are encoded with reference to yet other reference frames.

An operation of encoding frames will now be described beginning with theencoder in an initial state. Initially, frame buffer 230 and framebuffer 232 are empty. Frame loader 202 loads frame buffer 230 with aframe of the input video. Since there is no reference frame at thispoint, that frame in frame buffer 230 would naturally be encoded as akey frame. However, it should be understood, that in some variations, areference frame might be preloaded, in which case the first frameprocessed does not need to be the key frame.

Continuing the description of the operation, frame compressor 204obtains the frame from frame buffer 230 and compresses it into anencoded frame. Such a compression could be lossy or lossless (which,technically, is just a special case of lossy compression). That encodedframe is then provided to output scheduler 210 to form the output videosequence. The encoded frame is also provided to a frame decompressor 240that decompresses the frame and provides it to frame regenerator 242.The output of frame regenerator 242 is stored in frame buffer 232 as thereference frame to be used for subsequent encoding steps. Of course, ifthe output of frame compressor 204 is known to be a losslesscompression, such that the outputted frame decompressor 240 can be anexact replica of the current frame, then frame decompressor and frameregenerator 242 can be eliminated and instead the contents of frame 230could simply be copied into frame buffer 232 once the current frame isencoded. Either way, once the current frame has been processed, frameloader 202 can load another frame into frame buffer 230 and that framewould become the current frame to be encoded. At this point, a referenceframe is available in frame buffer 232 and a process of encoding thecurrent frame while taking reference to the reference frame will now bedescribed.

Where the current frame is encoded with reference to the referenceframe, the use of frame compressor 204 is not required. Instead, motionmatcher 206 can operate on the current frame, the reference frame, andsegment data about the reference frame generated by segmenter 220, tooutput from motion matcher 206 kinetic information, which are stored inkinetic information storage 236. The operation of motion matcher 206 isdescribed in more detail below. The kinetic information output by motionmatcher 206 relates to changes in segments from the reference frame tothe current frame. In other words, the reference frame is segmented suchthat areas of the reference frame are associated with segmentidentifiers, thus resulting in segments having segment boundariesbounding pixels of the reference frame. These segments can be matched topixels in the current frame and kinetic information about the segmentscan be identified. Merely one example of information about a segmentmight be a determination that a particular segment of the referenceframe is suitably represented by a similar collection of pixels in thecurrent frame, possibly offset in location and/or color values. Once asmany segments to be matched are matched from the reference frame topixels in the current frame, the kinetic information associated with thecurrent frame can be provided to output scheduler 210 to form part orall of the encoding of the current frame, as well as being provided toresidue generator 208.

Residue generator 208 can then, from the kinetic information, thecurrent frame, and the reference frame, determine what differences wouldremain between the current frame and the reference frame after thekinetic information is applied to the segments of the reference frame.Such a residue might include changes in position, shape, or color valueof pixels associated with a segment that are not already accounted forin the kinetic information. Residue might also include exposed area. Anexposed area would occur, for example, where the segments representobjects in a scene and those objects are moving between the referenceframe and the current frame. If that were the case, there would be somepixels in the current frame that are not associated with any segment ofthe reference frame because the objects or portions of objectsrepresented by those pixels of the current frame were objects orportions of objects obscured by other objects in the reference frame.This is illustrated by FIG. 7, which shows exposed areas 704 resultingfrom the frame-to-frame motion of segments 702. Thus, the pixel valuesfor exposed areas, and other residues might form the residue data outputfrom residue generator 208 to storage 238.

Residue data 238 is provided to output scheduler 210 to form anotherpart of the encoding of the current frame. As should be apparent fromthis description, if the residue data is an exact representation of thedifference between the reference frame and the current frame after thekinetic information is applied to the reference frame, then the outputof output scheduler 210 contains enough information such that thecurrent frame could be exactly reconstructed from nothing more thanprior knowledge of the exact contents of the reference frame, thekinetic information relating the reference frame and the current frameand the resulting residue data. However, in some cases exact replicationof the current frame is not always required, in which case residue data238 might be not the exact difference. If that is the case, then frameregenerator 242 is preferably used to regenerate the current frame fromthe reference frame, the kinetic information, and the residue data, sothat the reference frame used for subsequent encoding is not the exactreference frame, but the reference frame as it is known to berecoverable at the decoder.

Encoder 100 can repeat a process with subsequent frames becoming thecurrent frame, until the video is completely encoded. Although anencoder might always have a reference frame available, the encoder couldchoose to ignore the reference frame and encode the current frame as akey frame. This might happen, for example, as the result of an externaltrigger, upon detection of a scene change, or after the encoder hasdetermined that the residue data is such that encoding the current framewith the key frame would be more efficient. In instances where framesare not always losslessly compressed, the encoder might decide not touse the reference frame if it determines that sufficient losses haveaccumulated in the process of encoding frames and using those frames asreferences for subsequent encodings of frames that are used forreferences, etc.

FIG. 3 is a diagram illustrating the structure of a compressed videostream, as might be output by the encoder shown in FIG. 2. Asillustrated there, a frame K, followed by several non-key frames, suchas frame K+1 and K+2. As illustrated, frames K+1 and K+2 can be fullyrepresented by kinetic data, model data (explained in further detailbelow with reference to FIG. 6) and residue data. The kinetic data isshown, by way of example, is further detailed as comprising dataelements associated with segments of the current frame's referenceframe. In this example, the reference frame for frame K+1 might be frameK.

As an example of the kinetic information associated with each segment,the data elements there shown include translation data, z-order data,affine data, non-linear data, lighting data, and other data. The encodedvideo data stream might also include, either in a header applicable toall frames, or on a frame-by-frame basis, an indication of which of aplurality of segmentations schemes was used, partial segmentationinformation or segmentation hints and/or partial canonical informationabout how the segments are ordered or labelled with index values.Typically, canonical information is not needed in the compressed videoas the decoder should normally be able to order segments in the same wayas the encoder did. The encoded video data stream might have some of thekinetic information associated with segments in a segmentation omittedif it can be predicted by the decoder.

The encoding of difference between rough frame and raw frame can be anovel residue frame as described herein or just a simple differenceframe conventionally compressed. The residue frame can be encoded as aframe or might be encoded as segment-by-segment residue.

FIG. 4 illustrates additional data constructs that might be present inthe compressed video information. The additional information shown thereincludes a field for indicating the segmentation scheme used for thecurrent frame, partial segmentation information and/or hints aboutsegmentation usable by a decoder, canonicalization informationindicating an ordering of the segments and other hints that mightpossibly be present.

One canonicalization approach is to assign segment indices to segmentsbased on position in the frame. For example, the segment that includesthe pixel in the upper left corner of the frame could be segment 1, thesegment containing the next leftmost pixel in the top row that does notbelong to segment 1 could be segment 2, and so on through all the rowsof pixels. If this approach is used, the encoder and decoder canindependently determine the same canonicalization as they segment theirown copies of a reference frame.

FIG. 5 is a block diagram of a decoder according to embodiments of thepresent invention. As shown there, a decoder includes an input scheduler502, a frame decompressor 504, a segmenter 506, a current framereconstructor 508 and a processor 510. Input scheduler 502 receivescompressed video information, from a channel, from storage or fromanother source. For key frames, the video data can be provided to framedecompressor 504 for decoding. Frame decompressor 504 can then decodethe frame and store it in a frame buffer 520. For nonkey frames, thevideo data can be provided to kinetic information storage 522 andresidue storage 524. Other storage shown includes segment datasetstorage 526 and approximation frame storage 528.

In operation, when a key frame is received, input scheduler 502 providesit to frame decompressor 504, which decompresses the key frame, storesit in frame buffer 320 and that uncompressed frame can be output, forthe use of the video user coupled to the decoder. When a nonkey frame isreceived, other elements of the decoder process the frame. In someembodiments, the decoder might determine that the next frame is a keyframe by examining a flag in the compressed video data associated withthe frame.

Once a frame is decoded and output, it can be the reference frame,stored in frame buffer 520. The decoder includes a segmenter 506 thatcan segment the frame in frame buffer 520 into a set of segments. Thesegmentation results are stored in as a segment dataset in storage 526.There are many ways to structure the results. One such method is toidentify each segment with an index and a segment boundary, which is aclosed shape that encloses one or more pixels of the reference frame(although degenerate, zero-pixel segments should not be ruled out).Another method is to associate each pixel in the reference frame with asegment index. However it is stored, it should be noted that the decodercan generate the segment dataset, at least approximately, withoutrequiring any additional data from the encoder, which might increase thesize of the compressed video data. Thus, the decoder doing its ownsegmentation allows for greater compression than if the decoder reliedon the encoder's segmentation results.

When a nonkey frame is processed, the reference frame for that nonkeyframe is present in frame buffer 520. As explained above, a nonkey framemight have been encoded with reference to more than one reference frame,but for clarity, this explanation relates to the case where only onereference frame is needed for decoding a nonkey frame. It should benoted that the reference frame need not be the frame immediately priorto the nonkey frame in the video sequence and need not even be prior tothe nonkey frame being decoded.

As illustrated in FIG. 3, a nonkey frame is encoded by kinetic data,residue data and possibly other data. In the decoder shown in FIG. 5,the kinetic data for the nonkey frame is supplied to kinetic datastorage 522 and the residue data for the nonkey frame is supplied toresidue storage 524. The current frame reconstructor is coupled toreceive all or part of the reference frame information from frame buffer520, all or part of the segment dataset for that reference frame, andall or part of the kinetic information for the current frame. Thecurrent frame reconstructor is configured to generate a rough frame,stored in storage 528, from that information.

The rough frame is an approximation, although it might be exact, of thecurrent frame from the reference frame, its segmentation and kineticinformation relating the segments of the reference frame to the currentframe. Note that the segmentation information was not required to beincluded in the overhead of the compressed video, but instead could havebeen generated entirely by the decoder. In some embodiments, decodereffort might be more of a concern than efficient bandwidth usage, inwhich case the encoder might include in the compressed video somepartial segmentation information or hints to assist the decoder ingenerating its own segment dataset.

Processor 510 is configured to accept the rough frame and residueinformation to form a regenerated current frame, which can then beoutput by the decoder. The regenerated current frame might then be usedas a reference frame for later received (but not necessarily later inthe video sequence) frames.

Each of the components shown in FIG. 5 might be implemented in specialpurpose hardware, programmable hardware or software. For example, eachof components 502, 504, 506, 508 and 510 might be portions of oneprogram operating in an input data stream. Each of the storage elements520, 522, 524, 526 and 528 might be separate storage areas, or might beseparate portions of a common storage or memory. In some cases, where itis more efficient, the frame buffers might change roles rather thanhaving the data from one frame buffer copied to another frame buffer.

Generally, the operations of the components of the decoder perform theinverse of the operation performed by the encoder. For example, wherethe residue data is simply the a compressed difference frame of thedifference between a rough frame and the current frame, processor 510might simply read the residue data for the current frame, decompress itand add it back to the rough frame to result in a reconstruction of thecurrent frame.

Further compression might be possible by special coding of the kineticinformation as well as modelling the rough residue representing thedifference between the rough frame described above and the currentframe. For example, without further processing, the rough residue mightcontain data about exposed areas. A frame is an image of a real orgenerated scene and typically contains objects. In some segmentationschemes, segment boundaries follow boundaries of objects in the scene.If relative motion of an object is present between a current frame and areference frame, there will likely be a portion of the current framethat represents an object or background that does not correspond to asegment in the reference frame because that object or background wasobscured by another object in the reference frame but not in the currentframe. That area is referred to herein as an “exposed area.” An exampleof exposed areas is illustrated in FIG. 7.

FIG. 6 is a block diagram of a portion of a decoder that models therough residue to form model data and a remaining residue that ishopefully more compressible than the rough residue. As shown in thatfigure, the kinetic information is provided to a modeller 602 that alsohas access to the current frame, the reference frame and the segmentdataset of the reference frame. Modeller 602 generates model data 606that is output as part of the compressed data stream (see FIG. 3, for anexample of placement) and is provided to a residue generator that wouldgenerate the remaining residue.

To further compress the compressed data stream, a motion vector coder604 codes the motion vectors (and possibly other kinetic information) toreduce redundancy in the motion vectors, prior to the information beingincluded in the output video data. Referring now to FIG. 8, a flowchartof an encoding process is there shown. The process begins with receivinga key frame (step S1) and compressing the key frame to form an encodedkey frame that is output as the output of the encoder (S2). Thecompression can be lossy or lossless (as used herein, lossy compressioncan include zero loss (lossless) compression). If the compressionresults in loss of information (S3), the encoded key frame isdecompressed to form a reconstructed key frame to be used in subsequentsteps in place of the key frame. This allows the decoder to follow alongwith the encoder's coding process without the encoder having to conveyall of its state, because the encoder will operate on what the decoderhas, not what the encoder has (although in the lossless case, those willbe the same.

At step S5, the frame is now considered a reference frame and asegmentation is generated (S5). Segmentation can be done using knownmethods. Some approaches to segmentation are shown in U.S. Pat. No.6,778,698 (U.S. patent application Ser. No. 09/591,438 filed Jun. 9,2000 and entitled “Method and Apparatus for Digital ImageSegmentation”), which is commonly owned with the present application andis incorporated herein for all purposes. In some cases, an encoder mightselect among a plurality of segmentation schemes, so the encoder selectsa scheme. If the scheme is determinable from information that theencoder knows the decoder has, such as the content of prior processedframes, the encoder need not include an indication of the schemeselection in the output video data. The scheme selected might depend onthe image content, as some schemes might work better than others for agiven image.

At step S6, the encoder receives a second frame that becomes the currentframe. Here we assume that the second frame is not a key frame. If itwere, the process would loop back to step S1. If a key frame following akey frame is detected early enough, the segmentation of the first keyframe might be omitted if it would not get used as a reference for anynonkey frames. Note that the first frame and second frame need not beconsecutive and the first frame need not precede the second frame in avideo sequence.

Since we assume that the current frame is a nonkey frame, it isprocessed as such. First, segments of the reference frame (the key framedescribed above or a nonkey frame from a prior loop) are matched to thepixels of the current frame (S7) to form a segment mapping. The currentframe need not be segmented at this point—the mapping is from segmentsof the reference frame to pixels of the current frame. The process ofmotion matching might be performed in one or more methods described inU.S. Pat. No. 6,584,213 (U.S. patent application Ser. No. 09/912,743filed Jul. 23, 2001 and entitled “Motion Matching Method”), which iscommonly owned with the present application and is incorporated hereinfor all purposes.

Next, kinetic information for the segments of the segmentation isgenerated (S8). The kinetic information for a segment can be simply amotion vector representing an (X, Y) translation of a segment betweenthe reference frame and the current frame, but might include moreinformation. For example, the kinetic information for a segment mightindicate other information about the segment between the reference frameto the raw second frame, where the changes might include an indicationof a z-order of the segment (relative or absolute; determinable byexamining changes in the segment from frame to frame), deformation(rotation, dilation, other affine transformation or a nonlineartransformation defined by a set of deformation parameters), lightingchanges (an additive offset in one, two or three color planes, such asan additive offset in a luminance plane and/or a multiplicative offsetin one, two or three color planes), and/or residue by segment, or pixelcolor value offset (linear or nonlinear), such as a color offset for thesegment and a multiplicative offset for segment. While Bordering mightbe considered an characteristic of a specific image rather than anindication of the changes in a segment from one frame to the next, here“z-ordering” refers to z-ordering as determined by examining the changesof two segments relative to each other from one frame to the next.

Once the kinetic information is generated, a rough frame can begenerated (S9). A rough frame is the frame that would result by applyingthe segments of the reference frame generated in step S5 and the kineticinformation generated in step S8 to the reference frame. The roughframe, or the difference between the rough frame and the current frame,can be further processed to determine model data, as might result fromexposed area processing and applying non-motion related kineticinformation. In some embodiments, the model data is not generated orused.

Whether model data is used or not, the remaining difference between therough frame and the current frame is generally referred to herein as theresidue. A residue frame is generated (S11), if not already available,from the current frame by subtracting out the image portions or pixelvalues represented by the kinetic data applied to the segments of thereference frame and then subtracting out the image portions or pixelvalues represented by the model data, if used. Alternatively, theresidue frame could be generated by subtracting the rough frame from thecurrent frame.

This residue frame is compressed (S12), and if the compression is notlossless (S13), the resulting compressed residue frame is decompressed(S14) for use in later steps. If the compression is lossless, thecompressed residue frame does not need to be decompressed, as theuncompressed residue frame could be used in the later steps. In somecases, these steps could be omitted regardless of how the compression isdone, but preferably the later steps wherein the residue frame is usedto generate the reference frame used for later compressions would usethe residue frame as it would exist at the decoder, even if that is notexactly what the encoder started with.

Once all of that is done, the encoder can output the compressed currentframe as a compressed nonkey frame comprising the set of kineticinformation, model data (if used) and a compressed residue frame (S15).Then, the encoder determines whether the next frame will be a key frame(S16). This decision could be made based on some external trigger, adetermination that the current frame is from a different scene than thereference frame (scene change detect), or based on the results ofcompressing the current frame. Although not shown in the figure, theprocess might include further logic to discard the compressed nonkeyframe generated for the current frame if the compression is not goodenough and repeat the process with the current frame being treated as akey frame.

If the next frame is a nonkey frame, the current frame is labelled asthe reference frame (possibly moved into a frame buffer allocated forthe reference frame). Where the compression is not lossless, preferablythe decompressed current frame is used as the reference frame instead ofthe original uncompressed current frame, so that the encoder and thedecoder are in sync. The process then continues, looping back to stepS5, where the new reference frame is segmented and another frame isreceived, to become the now current frame. In some embodiments, thesubsequent frame uses a frame other than the immediately prior currentframe as its reference frame. In some embodiments, more than one priorencoded frame is used as the reference.

If the next frame is to be a key frame, the process loops back to stepS1 and repeats from there, with the next frame being the current frame.The process can loop until there are no more frames to encode. It shouldbe understood that the encoder might also perform steps such as addingheader information, such as at the beginning of a sequence or the end ofa sequence.

Referring now to FIG. 9, a flowchart of a decoding process is thereshown. The process begins with receiving a compressed frame (step S30).The decoder then determines if the received frame is a key frame (S31),such as by inspecting a “keyframe” flag in the input data.Alternatively, the decoder can guess from the prior decoded data usingrules known the encoder.

If it is a key frame, it is decompressed (S32), output (S33) and storedas a reference frame. If the decoder can determine that the currentframe will not be used as a reference frame, then it need not be storedas the reference frame. In some implementations, the frame does not haveto be stored and its current location is just labelled as the locationof the reference frame. In other implementations, the frame is movedinto storage specifically for reference frames. The process then loopsback to step S30 awaiting another frame.

If the received frame is not a key frame, the decoder receives kineticinformation (S35). If the encoded stream includes them, the decoderreceive an indication of which of a plurality of segmentations was usedby encoder, segmentation hints, partial segmentation information,canonical information or the like. From the received information, afirst rough frame is generated (S36). One way to generated the firstrough frame is to start with a blank frame and populate it with pixelvalues from segments of the reference frame, adjusting the segments asindicated by the kinetic information and other received information.

Next, the model data is received (S37) and a second rough frame isgenerated (S38) by modifying the first rough frame according to themodel data. Alternatively, the model data might have been receivedbefore step S36. The decoder might combine the steps of generating thefirst and second rough frames into one step of frame generation. Themodel data, if used, might include results of exposed area processing,general image parameterization or application of non-motion relatedkinetic information.

Next, the compressed residue frame is received (S39) and a reconstructednonkey frame is generated (S40) by modifying the second rough frameaccording to the residue data. As with the above steps, the ordering ofthe receipt and generation might be different than this exampledescribes.

Once the nonkey frame is generated, it can be output (S41). If thatframe is to be used as a reference frame for other nonkey frames, it isstored as the reference frame and segmented (S42) and the process loopsback to step S30 to receive the next frame until all the frames arereceived.

Specific Implementations

The examples described above can be further particularized depending onneeds of the application or known characteristics of the images. Thissection describes various optional implementation details that might beused to improve upon the above-described systems for particular needs orpurposes.

Partial Synchronization

In the examples above, it is generally assumed that the encoder and thedecoder are in sync. For example, where the encoder needs to make acoding decision, the encoder bases the decision on the version of thevideo sequence that the decoder has, not what the encoder has, if thosetwo differ. In a partial synchronization system, the encoder mightsometimes use information that is not available to the decoder and thedecoder would make a best guess. This is most useful where the effectsof a wrong guess at the decoder are not expected to be significant andthere are great gains to be made in computing effort or compressionratios if complete synchronization is not enforced.

Hints

Where the encoder determines that the errors between an original frameand the decompressed version of the compressed original frame, due tothe compression not being lossless, and that the errors would introduceundesirable artifacts, the encoder can include hints usable by thedecoder to reduce those errors. In a simple example, the encoderlosslessly encodes the error (which might require more bits overall thanif the original frame were losslessly compressed). In a more likelyexample, the encoder might determine which of the differences causesignificant segmentation differences and hint to correct for thosesegmentation differences. Such hints might allow the decoder toreconstruct (and thus use) a segmentation of the original frame ratherthan the reconstructed frame.

The hints might also be usable by the decoder to partially or fullysynchronize the decoder to the encoder segmentation or to the encodercanonicalization.

Metadata

Other information included in the compressed video data might bemetadata associated with segments. Since the segments are oftenassociated with objects in a scene, the metadata could be used forpoint-and-click operations. For example, the video stream might includereferences to areas of the input frame that are associated withmetadata. The encoder's segmenter can then associate that metadata withspecific segments. Those associations can be provided to the decoder,for use by the video user. For example, areas of the image could beassociated with URLs linking the area to a resource on the Internet.Those associations would be carried with the image as the video sequenceis presented to the video user.

Metadata is associated with segments and can remain with segments fromframe to frame as the segment evolves. For example, a segment's segmentindex may change due to changes in position or other canonicalizationprocess, but the metadata will remain associated with that segment(i.e., inherited from prior frame's segments). Metadata can beassociated with a group of segments and if a segment breaks up, the newmultiple segments will all inherit the prior frame segment's metadata.

Hierarchy of Segmentation

Where a frame is an image of a scene containing objects larger than apixel extent, the segmentation of the frame might tend to follow boundsof the objects. In the simplified example of FIG. 7, where the objectsare a truck with windows and tires, and a sun and a road, thesegmentation follows the bounds of the objects in the scene.

As might be apparent from FIG. 7, if the kinetic information for eachsegment is encoded separately, some redundancy will occur, as thedisplacements of the truck, the tires and the windows will typically bethe same or very similar. Reducing this redundancy would result in asmaller compressed size for the compressed video data. One way to reducethis redundancy is to establish a hierarchy of segmentation.

Using FIG. 7 as an example, the truck, tires and windows might begrouped into one higher level segment comprising several (five, in thiscase) segments at a lower level in the segment hierarchy. When codingsegment information such as translation, the high level segmenttranslation could be given, along with relative translations of thelower level segments relative to the “group” translation. This mightalso apply to other kinetic information, such as lighting changes.

Segmentation can be either bottom-up or top-down. With bottom-upsegmentation, an image is segmented based on image content intofirst-level segments (small segments) and those first-level segments aregrouped into second-level segments based on first-level segment content(i.e., pixel color values of pixels in the first-level segments,relative motion, etc.). The process is then continued to findthird-level segments up to N-level segments, if more than two levels ofthe hierarchy are used.

With top-down segmentation, an image is segmented based on image contentinto first-level segments (large segments) and those first-levelsegments are segmented into one or more second-level segments based onfirst-level segment content and the process is then continued to findthird-level segments down to N-level segments, if more than two levelsof the hierarchy are used.

For a bottom-up segment hierarchy, the generation of second-levelsegments might consider the boundaries of the first-level segments aswell as pixel values in the first-level segments.

A degenerate hierarchy (i.e., a one-level hierarchy) might result fromeither of these hierarchy approaches. Preferably, the segments that aregrouped into a group of segments are “simply connected” together.

An encoder can send the segment hierarchy to a decoder for use indecoding the frame, but in some embodiments, the encoder and decoder arein sync, so the decoder can generate a segment hierarchy from alreadydecoded frames for use in decoding a current frame.

The segment hierarchy might be useful in editing or formatting one ormore frames, wherein edits or formats are applied at varying levels ofgroupings of segments. The segment hierarchy is likely to be useful forcompressing the video data and/or associating metadata with elements ofthe frame, since the hierarchy of the segments might follow thehierarchy of objects in the scene. For example, for a scene of afootball game, higher-level segments might envelop a player and afootball, while lower-level segments envelop various areas of theplayer's jersey and helmet, the laces of the football, etc.

Motion Vector Grouping/Kinetic Data Elements Grouping

Instead of sending encoding each motion vector and each other kineticdata element separately, motion vectors (for example) might be groupedin various levels of hierarchy. The groupings can be done as describedherein for segment groupings, or the motion vectors can be groupedindependently of any segment grouping.

When a set of motion vectors is encoded as a group, the encodingincludes a group motion vector and residual motion vectors indicatingchanges of each member relative to their group motion vector. This canalso be done for other kinetic data elements, such as lighting changes.

The grouping of a current set of motion vectors can be done by lookingat a prior set of motion vectors and/or by looking at the content of thereference image, such as its segmentation. The reference image is theimage to which the motion vectors relate, such that the motion vectorsrepresent kinetic differences between the reference image and thecurrent image.

In some embodiments, the ordering of segments and the encoder anddecoder need not be consistent and the grouping is done by consideringthe values for the prior set of motion vectors or the segments of thereference image and group based on their values instead of any index.The indication of the grouping by value can also be partial, where theencoder conveys part of the grouping and the decoder determines therest.

In some embodiments, there is a distinction between the case where thereis no prior history and where there is prior history. In the formercase, the encoder encodes and the decoder decodes using groupings ofmotion vectors that are determinable from the contents or segments ofthe reference frame, while in the latter case the encoder can encode andthe decoder can decode using groupings of motion vectors that are basedon previously occurring motion vectors and/or previous groupings ofmotion vectors.

The reference frame can be a key frame or a previously coded/decodednonkey frame. In a video sequence, the current frame need not occurbefore or after the reference frame. The decoder assumes the motionvector data comes in groups and the decoder knows the hierarchy ofsegments. Thus, the encoded stream can be first level resolution ofmotion vectors, followed by residues as the next level, etc.

Synchronous Temporal Predicting of Motion Vectors at Encoder and Decoder

If the motion vectors (and other kinetic information) are encodedwithout reference to predictable changes, further compression might bepossible by omitting predictable changes, e.g., by only encoding thedifference between a predicted change and an actual change. For example,if a segment is moving from frame to frame in a linear direction, aprediction would be that the motion vector for subsequent frames wouldindicate continued movement in that linear direction. Thus, where theencoder and the decoder are both performing the prediction, the encoderneed only encode any variance from that linear motion as opposed toencoding the entire motion. The prediction can use a hierarchy ofsegments and/or a hierarchy of motion vectors, possibly resulting in ahierarchy of predictions.

One type of prediction is to examine the motion of segments in a firstframe and a second frame and predict that inertia will apply tomovements of the segments in a third frame. Another type of predictionis that if the light is fading linearly (or in a nonlinear, butpredictable way), the kinetic information about lighting is predicted.

In one variation, motion vectors are grouped and the group motionvectors are predicted while the residual relative motion vectors or eachmember of the group are encoded as the difference between the predictedgroup motion and the actual individual member motion, regardless of whatthe actual group motion is, thus encoding the correction to theprediction and the refinement of individual member movement.

Exposed Area Filling Performed by Encoder and Decoder

Where motion vector coding is used to encode a nonkey frame, exposedareas might be present in the resulting intermediate construction of thenonkey frame following application of motion vectors and other kineticinformation. The intermediate construction, or “rough” frame” is theframe generated, or generatable, from just a reference frame and a setof motion vectors describing the position changes of blocks or segmentsbetween the reference frame and the “current” frame being encoded ordecoded. If all of the segments in the reference frame find matches inthe current frame and none of the segments moved, then of course therewould be no gaps in the content of the current frame. However, if thereis motion at all from frame to frame, there are likely to be areas ofthe current frame that are “exposed” areas. Without more understandingof the objects in the scene being captured by the images formed by theframes being encoded, the pixel color values for pixels in exposed areasof the current frame cannot be deduced from the reference frame, becausewhatever object or object portion that is represented by the exposedarea was not present in the reference frame.

With understanding of the objects in the scene, however, the encoder anddecoder can infer what the newly exposed area looks like. For example,if it is apparent that the scene is a round ball passing over acheckerboard pattern, the encoder can infer that any areas that wereobscured by the ball in the reference frame but are exposed in thecurrent frame can be described by a continuation of the checkerboardpattern. This information can be used by the decoder to decode exposedareas without information from the encoder and the encoder can use thisinformation to determine what to omit from the encoded video datastream. In some cases, such as were the encoder estimates what theexposed area looks like and the estimate matches what is actually therein the current frame, the encoded video data need not include anyfurther information about the exposed area and the decoder will stillcorrectly construct the exposed area in the current frame.

The exposed area filling process described herein can be used in asegment-based system where the current frame is roughly encoded byreference to segments of a reference frame and motion vectors associatedwith those reference frame segments, but can also be used in ablock-based system where the current frame is roughly encoded byreference to blocks of a reference frame and motion vectors associatedwith those reference frame blocks. One difference between asegment-based system and a block-based system is the a segment-basedsystem uses frames divided up based on frame pixel color values, whereasa block-based system typically divides up a frame in a fixed mannerregardless of content, such as dividing up each frame into 8 pixel by 8pixel macroblocks.

Typically, in a block-based system, the blocks are matched and motionvectors generated and the leftover areas of the current frame (theexposed areas) are encoded as part of the residue. This can beinefficient, if the residue contains bits of information that thedecoder can infer by itself from what information the decoder alreadyhas.

In a typical segment-based process, a key frame is received, compressedand used as a reference frame. If the compression is not lossless, thecompressed frame is decompressed and used instead of the original. Theframe is then segmented.

When a nonkey frame is received, it is processed with reference to thesegmentation and content of the reference frame. Thus, segments arematched between the segmentation of the reference frame and the contentof the current frame. The kinetic information describing changes in thesegments from frame to frame is then generated. From this, a firstapproximation to the nonkey frame is generated or exists.

The pixels of the current frame that are not associated with segmentsfrom the reference frame are considered exposed areas. These exposedarea pixels can be assigned to new segments, as appropriate. Each of thenew segments is then processed to determine how to code the pixels fromthe current frame that fall into these new segments. Examples aredescribed below.

For decoding, the decoder can generate the first approximation from thereference frame and the motion vectors. It then applies the encoding ofthe exposed areas, as provided by the encoder, to fill in the rest ofthe frame. In some cases, the exposed area filling information leavesnothing to be included as residue, but in other cases, the result ofapplying the exposed area filling is a second approximation of thecurrent frame that is then processed according to the residue data.

One method of exposed area filling examines the pixel color values ofadjacent segments and continues the color values or patterns of adjacentsegments with higher z-order values than other adjacent segments. Thiswould be an accurate reconstruction of an exposed area where one objectin the scene is partially obscuring another object in the scene. Theobscured object would have a higher z-order, because it is further fromthe point of view of the frame and thus it would be a good estimate toassume that when the closer object moves to expose an exposed area, theexposed area is actually part of the obscured object. If that doesn'thappen to be the case, then the difference can be coded as exposed areafilling, or it can be ignored and be included as part of the residue.

Note that the encoder and decoder can both perform the exposed areafilling process, so that the encoder does not need to encode the resultsof the process, other than the variance between the current frame andthe results of the process. In cases where one of a plurality of fillschemes is used, the encoder can encode its selection, unless thedecoder can infer correctly the scheme used.

Exposed area fill information might include bounds of areas to befilled, and/or pixel values of pixels in areas to be filled. Partialfill information or fill hints might be included in the encoded videodata and the exposed area filling process need not be exact, ifdifferences in the filling done by the decoder and encoder areacceptable. In some implementations, the exposed area filling is doneentirely at the encoder and conveyed to decoder, in which case, decoderprocessing is not required.

Residue Coding Performed Using Basis Functions by Encoder and Decoder

In some of the examples described above, after various coding steps areperformed, the remaining residue between the current frame and what hasbeen coded is then just coded as a difference frame using conventionalmethods. For example, since it is expected that most residue frameswould be sparsely populated, it can be quickly compressed usingrun-length encoding. However, in some cases, compression of the residueframe can be improved by coding the residue using basis functions. Thus,the encoded residue would comprise coefficients of basis functions knownto the encoder and the decoder. Basis function encoding of a residueframe could be done where the residue is left over after a segment-basedcoding, or left over after other types of coding.

1. A method of encoding uncompressed video data to form compressed videodata wherein the video data comprises a plurality of image framesincluding a set of reference frames and a set of nonkey frames, whereinthe set of nonkey frames are encoded with reference at least to asegmentation of at least one of the image frames in the set of referenceframes, and further wherein the segmentation is an assignment of some orall of the pixels of the reference frame to at least one segment basedon at least one of a color value or a location of the pixels in thereference frame, the method comprising: encoding a first referenceframe; segmenting the first reference frame into a first set ofsegments; segmenting the first reference frame into a second set ofsegments, wherein the first set and second set form a hierarchy ofsegments and each segment encloses a simply connected set of pixels ofthe first reference frame; matching segments of the hierarchy ofsegments to pixels of a nonkey frame; encoding into the compressed videodata a representation of the nonkey frame according to results of thestep of matching.
 2. The method of claim 1, further comprising encodinginto the compressed video data a representation of the hierarchy ofsegments, for use in decoding the nonkey frame.
 3. The method of claim1, further comprising steps of segmenting the first reference frame intoa third set of segments.
 4. The method of claim 1, wherein segmentingthe first reference frame into a second set of segments comprisesgrouping segments from the first set of segments such that each segmentof the second set of segments comprises one or more segments from thefirst set of segments.
 5. The method of claim 4, wherein the grouping ofsegments is performed based on pixel color values of pixels in segmentsof the first set of segments.
 6. The method of claim 4, wherein thegrouping of segments is performed based on boundaries of segments of thefirst set of segments.
 7. The method of claim 4, further comprisingsteps of segmenting the first reference frame into a third andsubsequent sets of segments wherein segments at each level of thehierarchy of segments are groupings of segments of its immediately priorlevel.
 8. The method of claim 1, wherein segmenting the first referenceframe into a second set of segments comprises dividing segments from thefirst set of segments into one or more segment of the second set ofsegments such that each segment of the first set of segments comprisesone or more segments from the second set of segments.
 9. The method ofclaim 8, wherein the division of segments is performed based on a colorvalue of pixels in segments of the first set of segments.
 10. The methodof claim 8, further comprising steps of segmenting the first referenceframe into a third set of segments wherein segments at each level of thehierarchy of segments are divisions of segments of its immediately priorlevel.
 11. The method of claim 1, further comprising encoding thecompressed video data without a complete representation of the hierarchyof segments, such that a decoder would have to independently generate atleast a portion of the hierarchy of segments.
 12. The method of claim 1,wherein the hierarchy of segments is usable for editing one or moreframes.
 13. The method of claim 1, wherein the hierarchy of segments isusable for formatting one or more frames.
 14. The method of claim 1,wherein the hierarchy of segments is usable for compressing one or moreframes.
 15. The method of claim 1, wherein the hierarchy of segments isusable for associating metadata with elements of the frame.
 16. Themethod of claim 1, wherein the hierarchy of segments is usable forcoding one or more of motion, residue or other kinetic information.